splunk > 


Splunk Cluster Administration - Class Lab Exercises 


Training Lab Environment 


Throughout the course, you will be working in a private network environment. Your instructor will provide 
the following information to configure and test your Splunk Cluster environment: 


e SWEDE OO Oaa ee yiatioaase sees ia iraa ia 
e Public address {Public_DNS} = aaaasannnnnnnsrrssnrnen C<session>-<x>.class.splunk.com 
e SSH user name {user} qj =  siiniscccbianmasshcarvdedesuncieencseeccesecedcnaadescsebeniceigdeaandaaiiernteeteers 
e SSH password {password}:  biitavedensdndeniimecsnceutndedeaseatacResicedadandsceairte des wenneewcesndandecgaddadace 
e — Splunk user name: > ‘wiscvasanesracissamgoneurtenanteuvaaunteiaiaannieicensimneitaleinseRieesiane: admin 
e Splunk user password: — lussarnrrerrrnrrerrrrrorernrrerrerrne same as SSH user password 


NOTE: Your SSH password and Splunk admin password are the same on all instances. 


You will use the Splunk CLI for configuration tasks and use Splunk Web for monitoring and verification 
tasks. To access your Splunk CLI consoles, you will first SSH into your designated Misc-Server address. 
From there, you will remote-SSH into other nodes in the network using their private IP addresses. 


S3 Remote 
Storage 





Your 
Computer 








10.0.x.2 
Search 
Head 

Cluster. 


nO 





SSH 10.0.x.2> 


Your Misc-Server is also configured to be the reverse-proxy web server for your Splunk Web instances. 
To access a particular instance, go to https://{Public_DNS}/{splunk server name}. For example, 
to access Splunk Web for cmanager, direct your web browser to https://{Public_DNS}/cmanager. 


t--> 1dX2 Web Ul 
<> idx3 disabled 


r--> idx4 < i 
--->cmanager 
Browser --->dserver 
--->sh1 


--->sh2 
+--> sh3 
=--> sh4 
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Module 1 Lab Exercise — Configure Splunk License Manager 


Description 


In this exercise, you will perform basic discovery tasks to learn about your lab environment and start the 
Splunk License Manager. 


Steps 
Task 1: Access your designated Splunk environment. 


1. SSH to the Misc-Server with the credentials provided. 


ssh os_user@{Public DNS of your Misc-Server} 
os user@Cnnnnn-x.class.splunk.com's password: {SSH user password} 





NOTE: Use the SSH password given to you at the beginning of the class. 


2. Check the prompt and verify your student ID and the host you are on: 
The third number of the prompt represents your student ID. (x in this example) 


The last number of the prompt indicates the current node you are on: 

e -1is your indexer cluster and referenced as IDX-Cluster in this lab exercise. 

e -2 is your search head cluster and referenced as SH-Cluster in this lab exercise. 

e -3 hosts miscellaneous servers and referenced as Misc-Server in this lab exercise. 


3. Type pwd to display your current directory and type 1s to list the instances on this server. 


pwd 


J SOr/ oeme OS Veet 


ls 





cmanager dserver fwdr 


Task 2: Set up password-less SSH connection to IDX-Cluster and SH-Cluster. 
4. For convenience, set up password-less SSH connections using ssh keys from your Misc-Server. 


NOTE: This allows remote-ssh to your IDX-Cluster and SH-Cluster without entering a password in 
this course. Your production environment might have a more secure implementation. 


5. Generate and share the public key with the IDX-Cluster and SH-Cluster nodes using scp. 
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ssh-keygen -t rsa -P "" 

Generating public/private rsa key pair. 

Bater ELIS la winch EO Sere the key | /opt/Incme/ OS MSe%,/ .SSla/i¢l ese) 
Press Enter to accept the default value 


Created irector § /Opir/ineme/CS USer/ esl" - 

Vous IGSTELELGe eon lias geen Serrecl Lm / OD) mome/;OS USS) .esla/acl se. 
vou pilico key aae Ceen sevel 1m /CweE/ mome/oOSs Meet oeni sa. ovo. 
The key fingerprint is: 

The key's randomart image is: 

E a (4) 9)e) | eae 


cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized keys 


chmod 600 ~/.ssh/authorized keys 


scp -r ~/.ssh 10.0.X.1:~/ 

Pheraurchent icity Your No- Caa TOTO I (IO) a eoce ll) |S ecole lees ToS se Ikat silavsiol a 
GIDE NN le 16 Ligle(sueenasigne ois) age 4 6) (sO) Siies well) Sic leslie Aelieley oc Rise | ey ZS siceu hie hele 
Are you sure you want to continue connecting (yes/no)? yes 

Warning: Permanently added '10.0.x.1"' (ECDSA) to the list of known hosts. 
os user@10.0.x.1's password: {use your SSH password} 

known hosts TOO 44a .4KB/s COS OC 

el e UD DO SSS . 4KB/s OORO 
sUtporized eyo 100% 398 .4KB/s CHO S 30) 

Cl Sel LCs Ieys .6KB/s 00:00 


scp -r ~/.ssh 10.0.x.2:~/ 

Mats: Vel bielisiue deine ne Ines VIO <a 2 (Nas, 4) | veciol ie lee ToS cel aned 
ECDSA ker EIn Or Pr Ine ois) age 4 EO sicis el SSclecly A elieley So Riso | oy Zia sic@u She ilely 
Are you sure you want to continue connecting (yes/no)? yes 

Warning: Permanently added '10.0.x.2' (ECDSA) to the list of known hosts. 
os user@10.0.x.2's password: {use your SSH password} 


From this point onwards, you should be able to ssh to each node without entering the password. 
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6. Remote-ssh to your IDX-Cluster, list the instances on the node, and exit back to Misc-Server. 


[os _user@ip-10-0-x-3 ~]$ 
ssh 10.0.x.1 


[os _user@ip-10-0-x-1 ~]$ 
ls 
wor ae E ex medion 4! 


exit 
Ogor 
Connection to IM) AOR al closed. 





7. Remote-ssh to your SH-Cluster, list the instances on the node, and exit back to Misc-Server. 


[os _user@ip-10-0-x-3 ~]$ 
ssh 10.0.x.2 


[os _user@ip-10-0-x-2 ~]$ 
ls 
shl sh2 sh3 AE 


exit 
Fogout 
Connection Lo 10.0.2 closed. 





Task 3: Configure your License Manager instance. 


8. In the Misc-Server session, start the dserver Splunk instance and check its servername, splunkd- 
port, and web-port. 
[os_user@ip-10-0-x-3 ~]$ 
~/dserver/bin/splunk start --accept-license 
This appears to be your first time running this version of Splunk. 


Nate oplunk web interla e ae eie Meco. / ao =< en 


~/dserver/bin/splunk show servername 
Splunk username: admin 


Password: {Splunk user password} 
Server name: dserver 


~/dserver/bin/splunk show splunkd-port 
Punk joxomes ile ® 


~/dserver/bin/splunk show web-port 
Web port: 8100 
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9. Configure the dserver instance as the License Manager for your deployment by adding the license 
file splunk. license. big.license from the /opt/license directory. 


~/dserver/bin/splunk add licenses /opt/license/splunk.license.big.license 
The licenses object has been added. You need to restart the Splunk Server 
(splunkd) for your changes to take effect. 


~/dserver/bin/splunk restart 





Check Your Work 


Task 4: Confirm the license information. 


10. Use your web browser to access the Splunk Web interface of your License Manager (dserver). 


https://{your assigned Public _DNS}/dserver 


NOTE: In this lab environment, a proxy web server in the Misc-Server has mapped all your Splunk 
Web instances to https://{Public_DNS}/{servername}. 


11. Log in as admin using your assigned Splunk password. 


NOTE: When you see the prompt Help us improve Splunk software, click OK. 
When you see the Important changes coming, click Don’t show me this again. 


12. Click Settings > Licensing and verify the information on the Licensing page. 





Licensing 


This server is acting as a master license server = Change to slave 
| 


Enterprise license group = Change license group 
This server is configured to use licenses from the Enterprise license group 
SE 
Pools Indexers Volume used today 
auto_generated_pool_enterprise | O MB / 200 MB <—= Edit | Delete 


No indexers have reported into this pool today 
© Add pool 


Local server information 


indexer name dserver <= 
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Troubleshooting Suggestions 


If your configuration is not returning the expected results, troubleshoot by isolating the issue. 
1. Verify the command syntax and spelling. 

[os_user@ip-10-0-x-3 ~]$ ~/dserver/bin/splunk btool check --debug 
2. Check splunkd. log for any errors: 


tail -50 ~/dserver/var/log/splunk/splunkd.1log 
Or, egrep 'ERROR|WARN' ~/dserver/var/log/splunk/splunkd. log 


3. Compare the output of ~/dserver/etc/system/local/server.conf with lab_conf_outputs.txt. 
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Module 2 Lab Exercise — Enable Single-site Indexer Cluster 


Description 


In this exercise, you will configure a Splunk single-site indexer cluster with three peer nodes and one 
search head. You will then simulate a peer node failover scenario. 


Steps 


Task 1: Configure the manager node for a single-site indexer cluster. 


1. In the Misc-Server ssh session, navigate to the cmanager Splunk instance and start Splunk. 


[os _user@ip-10-0-x-3 ~]$ 
~/cmanager/bin/splunk start --accept-license 


This appears to be your first time running this version of Splunk... 
rhen plun Wel a er aee ere Me eos / / yo s 2 S00) 





2. Check its servername, splunkd-port, and web-port. 


~/cmanager/bin/splunk show servername 
Splunk username: admin 

PES Swe 

server name: cmanager 


~/cmanager/bin/splunk show splunkd-port 
Plunk eor e oN eN 


~/cmanager/bin/splunk show web-port 
Web port: 8000 





3. Configure cmanager to be a license client to dserver (10.0.x.3:8189): 


~/cmanager/bin/splunk edit licenser-localslave -master_uri https://10.0.x.3:8189 
The licenser-localslave object has been edited. 





You need to restart the Splunk Server (splunkd)... 


4. Configure cmanager to be the manager node with the following indexer cluster options: 


e replication_factor 2 
e search_factor 2 
e secret idxcluster 


~/cmanager/bin/splunk edit cluster-config -mode manager -replication factor 2 
-search factor 2 -secret idxcluster 

The cluster-config property has been edited... 
You need to restart... 


~/cmanager/bin/splunk restart 





Task 2: Monitor the cluster status from cmanager's Splunk Web. 
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5. Access the Splunk Web interface for cmanager: https://{Public_DNS}/cmanager 
6. Log into Splunk Web as admin using your assigned password. 


7. Navigate to Settings > Licensing and review the information on the Licensing page. 


Licensing 


This server is associated with a remote master license server $ Switch to local master 


Local server information 
Indexer name cmanager 
Master server URI https://10.0.1.3:8189 
Last successful contact time 16 seconds ago (10/13/20, 4:34 PM) 


Messages Show all messages 


8. Navigate to Settings > Indexer clustering and monitor the cluster status. 


cmanager is listed as the only search head. The health status indicator is red and there is a 
message notification. 


Administrator v 1) Messages ¥ Settings 7 Activity v Help » Find 





Edit » More Info v Documentation [4 


Indexer Clustering: Master Node ) î 


No Peers Configured 


To learn how to configure peer nodes, refer to the documentation. Learn More [2 


Search Heads (1) 


filter Q 10 per page v 
i Search head name + Status > 
> cmanager é Y Up 


9. Click the Messages drop-down menu and read the message about the requisite number of peers 
required to join cluster. Acknowledge the message by deleting it. 


Administrator v 1) Messages ¥ Settings Y Act 





Å Waiting for requisite number of peers to join the X 
cluster. - https://127.0.0.1:8089. Cluster has only 
O peers (waiting for 2 peers to join the cluster). 
11/17/2020, 8:48:33 PM = 


Delete All 


ia 
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10. Click the red Health Status icon and drill down each red indicator to learn more about the status. 
Note that they are all related to health and status of the cluster. 


Health Status of Splunkd x 


E splunkd Ø Data Durability 


File Monitor Input e Root Cause(s): 


© BatchReader-0 e Replication Factor is not met 
@ TailReader-O e Search Factor is not met 
Index Processor e Last 50 related messages: 
@ Buckets e 11-17-2020 20:47:13.010 +0000 INFO CMMaster - event=service status=skipping 
reason='Cluster has only O peers (waiting for 2 peers to join the cluster). ' 
@ Disk Space e 11-17-2020 20:46:13.508 +0000 INFO CMMaster - event=service status=skipping 
@ Index Optimization reason='Waiting for the configured quiet_period=60 adding_peers=0' 
Indexer Clustering e 11-17-2020 20:46:13.077 +0000 INFO CMMaster - event=addSearchhead 
@ Cluster Bundles guid=10826AD5-1413-4165-9D01-B5BFFB1DA47D serverName=cmanager site=default 


E Data Durability hostPort=ip-10-0-1-3:8089 polllnterval=5.000 lastContactTime=1605645973.077655 


E Data Searchable 
@ Indexers 
Ø Indexing Ready 


~ f an 


11. Close the Health Status page but keep the Indexer Clustering: Master Node page in view. 


Task 3: Configure three indexers to form the replication peers for the single-site indexer cluster. 


Each indexing node in a production environment must run on a dedicated host. However, to simulate a 
working cluster in this lab environment, each Linux host has been configured to run multiple Splunk 
instances. To accommodate this, each instance has been carefully assigned unique port numbers. 


Reference the following port matrix when you configure each indexer instance: 


Server Name Splunkd-port Web-port Listening-port Replication-port 


8189 8100 9197 9100 


8289 8200 9297 9200 
8389 8300 9397 9300 





12. To form a cluster of indexing peers, remote-ssh to IDX-Cluster. 
13. Bring up idx1, idx2, and idx3 and perform the following: 
a. Configure each indexer peer as a License Manager to dserver. 
b. To receive forwarder data, configure a listening port on each peer. 
c. Tosave CPU resource, disable Splunk Web on each peer. 
d. Configure each instance to join the indexer cluster. 
e 


Monitor the Indexer Clustering: Master Node page as you bring up each node. 


NOTE: Be sure to use the same secret that the manager node has used. 


Defer restarts until you configure the cluster setting. 
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ssh 10.0.x.1 


~/1dx1/bin/splunk start --accept-license 

~/1idx1/bin/splunk edit licenser-localslave -master_uri https://10.0.x.3:8189 
Splunk username: admin 

Password: 


The licenser-localslave object has been edited. You need to restart the 
Splunk Server (splunkd) for your changes to take effect. 


~/1dx1/bin/splunk enable listen 9197 
Eeee aao ieue Syjollibliol< clevcel mim WGI joe Oy. 


~/idx1/bin/splunk disable webserver 

You need to restart the Splunk Server (splunkd) for your changes to take 

Si TOOT 

~/idx1/bin/splunk edit cluster-config -mode peer -master_uri https://10.0.x.3:8089 
-secret idxcluster -replication_port 9100 

The cluster-config property has been edited. 

You need to restart the Splunk Server (splunkd) for your changes to take 

Sr TOCE- 

~/idx1/bin/splunk restart 


14. Confirm the idx1 peer node has appeared on the cmanager's indexer clustering view. 


15. Repeat the steps to configure idx2: 


~/idx2/bin/splunk start --accept-license 

~/idx2/bin/splunk edit licenser-localslave -master_uri https://10.0.x.3:8189 
~/1dx2/bin/splunk enable listen 9297 

~/1dx2/bin/splunk disable webserver 


~/idx2/bin/splunk edit cluster-config -mode peer -master_uri https://10.0.x.3:8089 
-secret idxcluster -replication_port 9200 


~/1dx2/bin/splunk restart 
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16. Repeat the steps to configure idx3: 


~/1dx3/bin/splunk start --accept-license 


~/1dx3/bin/splunk edit licenser-localslave -master_uri https://10.0.x.3:8189 


~/1dx3/bin/splunk enable listen 9397 
~/1dx3/bin/splunk disable webserver 


~/1dx3/bin/splunk edit cluster-config -mode peer -master_uri https://10.0.x.3:8089 
-secret idxcluster -replication port 9300 





~/1dx3/bin/splunk restart 


Task 4: Configure a search head to join the cluster. 


17. Connect to the SH-Cluster session and identify what instances are on the host. 


[os _user@ip-10-0-x-1 bin]$ 
ssh 10.0.x.2 





18. To add a search head to the indexer cluster, bring up sh1: 
a. Configure it as a License Peer to dserver. 


b. Configure it to be the search head of the indexer cluster. 


NOTE: Don't forget to use the same secret that the Manager Node has used. 


[os _user@ip-10-0-x-2 ~]$ 

~/shi/bin/splunk start --accept-license 

~/shi/bin/splunk edit licenser-localslave -master_uri https://10.0.x.3:8189 
~/shi/bin/splunk edit cluster-config -master_uri https://10.0.x.3:8089 -mode 
searchhead -secret idxcluster 

~/shi/bin/splunk restart 

exit 


[os _user@ip-10-0-x-1 bin]$ 
exit 


[os_user@ip-10-0-x-3 bin]$ 





Check Your Work 


Task 5: Verify that your indexer cluster is functioning properly. 
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19. Access cmanager's Indexer Clustering: Master Node page and confirm the cluster status. 
e The Health Status indicator is green. 
e The message about the cluster has cleared. 


e The Indexer Clustering: Master Node page lists 3 peers and 2 search heads. 








splunk Apps ¥ Administrator v Messages v Settings ¥ Activity ¥ Help v Find 
Indexer Clustering: Master Node l | 
Edit v More Info v Documentation 2 
v All Data is Searchable vV Search Factor is Met Y Replication Factor is Met 
3 searchable o not searchable 2 searchable o not searchable 
Peers Indexes 

Peers (3) Indexes (2) Search Heads (2) 

filter Q 10 per page ¥ 
i Peer Name $ Fully Searchable $ Status $ Buckets $ : 
> idx3 v Yes Up 4 
> idx2 v Yes Up 8 
> idx v Yes Up 6 
NOTE: If any status indicator is NOT green, then check the configuration of the missing node(s) 


before you proceed to the next task. 


20. Open a new browser tab and go to: https://{Public_DNS}/sh1i 


Log into Splunk Web as admin using your assigned password. 
21. Navigate to Settings > Distributed search > Search peers. 
Notice the search peers are automatically added. 
22. Inthe Search & Reporting app, search index=_audit over Last 24 hours. 


a. Click host and see that the results contain events from idx1, idx2, idx3, and sh1. 


b. Click splunk_server and see that the results are from idx1, idx2, idx3, and sh1. 
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Configuration Troubleshooting Suggestions 


1. Verify the command syntax and spelling on each instance with: splunk btool check --debug 
2. Inthe manager node's Splunk Web, search for any cluster errors (adjust the time range): 


index=_internal sourcetype=splunkd (warn OR error) component=CM* 


3. Check splunkd. log of each instance for any errors: 
egrep 'ERROR|WARN' ~/<instance_name>/var/log/splunk/splunkd. log 


4. Compare the output of .conf files with Llab_conf_outputs. txt. 


S If you are done with the configuration and have about 15 
> minutes to spare, try this optional failover test. 





Optional Peer Node Failover Test Steps 


Task 6: Test a peer node failover scenario. 


23. From sh1, run the following search to establish a baseline search window (Last 24 hours): 


| makeresults 1 | eval custom_range="latest=".time() | table custom_range 


| makeresults 1 | eval custom_range="latest=".time() | table custom_range 
v 1 result (10/19/20 9:00:00.000 PM to 10/20/20 9:58:30.000 PM) No Event Sampling v 
Events Patterns Statistics (1) Visualization 


20 Per Page v» £ Format Preview v 


custom_range + 


latest=1603231110.988553 


NOTE: You cannot start a search with the eval command. To work around this limitation, use the 
makeresults command and create a dummy event. The above search generates a fixed 
search window that you use to validate the subsequent test. 


24. Copy the calculated search window value (custom_range) from the result. 


Example above: latest=1603231110.988553 
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25. Run a baseline search that displays the event count per host and its data provider: 


index=_audit host=idx* {copied value} | stats count by splunk_server host 


index=_audit host=idx* latest=1603231110.988553 | stats count by splunk_server host Last 24 hours ¥ ey 


v 15,926 events (10/20/20 9:06:07.000 PM to 10/20/20 9:58:30.988 PM) © Joby 2? ? Smart Mode 


No Event Sampling v 4 
Events Patterns Statistics (3) Visualization 


20 Per Page v af Format Preview v 


splunk_server + f host $ rá countè 7 
idx1 idx1 5317 
idx2 idx2 5317 
idx3 idx3 5292 


NOTE: Note the count values. Your result will vary. 


26. On cmanager's browser window, navigate to Settings > Indexer clustering and monitor the page. 


https://{Public_DNS}/cmanager/en-US/manager/system/clustering 
2/. To simulate a peer failure, connect to the IDX-Cluster ssh session. 


28. Identify the parent splunkd process of idx1 and stop the process. 


NOTE: _ This is only to simulate a peer failure in this lab environment. Your cluster peer nodes are 
running on a single host. 


ssh 10.0.x.1 


ps -ef | grep "splunkd -p 8189" 


user 25586 OT 00-12:47 splunkd —p sl39o > restart 
Usieie Zoos! 25536 0 2 WOS0NSIS [sjolltialcl joc 23586 sollwudlecl so Gilbsis ieSsiceicie 
[process-runner ] 





kill 25586 


29. In the cluster status page, wait until the status of idx1 becomes Down but all data is searchable. 


NOTE: The idx1 status should transition from Up > Pending > Shutting down while the cluster tries 
to recover. The search capability is ready when you see the green checkmark next to All 
Data is Searchable and the status of idx1 is Stopped. The fixup process begins and 
eventually the cluster will reach the complete state. 
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30. Re-run the exact search (on sh1) from Step 25. 


index=_audit host=idx* {copied value} | stats count by splunk_server host 


NOTE: The result still contains events from all peer nodes but the data providers are different now. 


v 11,431 events (9/25/18 9:58:37.000 PM to 9/25/18 10:54:46.947 PM) No Event Sampling ¥ ? Smart Mode v» 


Events Patterns Statistics (3) Visualization 


20 Per Page » # Format Preview ¥ 
splunk_server $ 
idx2 


idx2 





idx3 


The idx1 count doesn't exactly match the original count. Why? 
31. To restore the full cluster, start the Splunk instance for idx1 in the IDX-Cluster ssh session. 


~/1idx1/bin/splunk start 
exit 





32. Check the cluster status and verify that all peers are Up. 


33. Re-run the exact search again and verify the result. 


index=_audit host=idx* {copied value} | stats count by splunk_server host 


NOTE: The total counts for each host should now match your initial results. The search you ran 
while idx1 was down only includes replicated data from idx1, not the events that were 


indexed before you implemented indexer clustering. This search once again includes the 
non-replicated events from idx1. 


v 11,457 events (9/25/18 9:55:22.000 PM to 9/25/18 10:54:46.947 PM) No Event Sampling v ? Smart Mode v» 
Events Patterns Statistics (4) Visualization 


20 Per Page ¥ # Format Preview ¥ 


splunk_server $ 
idx1 
idx2 


idx3 


idx3 
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34. On the cmanager's Indexer clustering page, click the Indexes tab > the Bucket Status button > the 
Indexes With Excess Buckets tab. 


NOTE: The page lists the number of buckets exceeding the replication or search factor per index. 


Fixup Tasks - In Progress (0) Fixup Tasks - Pending (0) Indexes With Excess Buckets (2) 


Here is a list of indexes with buckets exceeding the replication or search factor. 


Remove All Excess Buckets 


Index Name $ Buckets with Excess Copies $ Buckets with Excess Searchable Copies $ Total Excess Copies $ Total Excess Searchable Copies $ Action 


_audit 1 1 


1 1 Remove 


_internal Remove 





35. To reclaim the storage space, click Remove All Excess Buckets > Confirm. 


36. Reload the browser page and verify that the excess copies are removed. 


Task 7: Investigate the peer outage with Splunk internal logs. 


From cmanager, search the metrics. log of idx1 to determine the time of outage (Last 60 minutes). 


index=_internal host=idx1 sourcetype=splunkd metrics | timechart count by host 





index=_internal host=idx1 sourcetype=splunkd metrics | timechart count by host Last 60 minutes v EA 


v 5,703 events (10/27/20 6:25:00.000 PM to 10/27/20 7:25:20.000 PM) No Event Sampling ¥ 


Job v eo 8 &£ €? Smart Mode v 
Events Patterns Statistics (61) Visualization 





+ Line Chart # Format 88 Trellis 





Oct 27, 2020 7:19 PM 
329 


6:27 PM 6:30 PM 6:33 PM 6:36 PM 6:39 PM 6:42 PM 6:45 PM 6:48 PM 6:51PM 
Tue Oct 27 


2020 


6:54 PM 6:57 PM 7:00 PM 7:03 PM 7:06 PM 7:09 PM 7:12 PM 7:15 PM 7:18 PM 7:21PM 7:24 PM 


time 


NOTE: Look for the drop-off in indexed events for idx1. 


37. Mouse over idx1's trendline at the point immediately before the dropoff, then click the square in the 
pop-up that displays the timestamp and count. 


A drilldown search executes and displays the events during that time period. 
38. Click the time range picker and select the Date & Time Range tab. 


39. Change the scope from Between to Since and then click Apply. 
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40. To display the fixup summary, search using your selected time range (Since date time): 


index=_internal sourcetype=splunkd_access uri_path="/services/cluster*" 
| timechart values(uri_path) by host 


NOTE: Depending on the time range of your search, a default span value is used to group the 
results. You can include the span option to adjust as necessary (span=1m for example). The 
results of this search cannot be charted on the Visualizations tab because you are creating 
multi-valued fields using the values function. To see the fixup tasks, look in the idx2 and 
idx3 columns. 


41. To identify the outage and recovery from the manager node, search using your selected time range: 


index=_internal sourcetype=splunkd component=CM* 


NOTE: To determine the time of outage, you can search for the event where the manager first 
detected the peer outage: (first entry) 


index=_internal sourcetype=splunkd component=CMMaster streaming error 


You can further investigate the cause by searching component=CMPeer: 


index=_internal sourcetype=splunkd component=CMPeer 
| timechart span=1m values(event_message) 


...transitioning from=Up to=Pending reason="non-streaming failure" 
...transitioning from=Pending to=Down reason="heartbeat or restart 
timeout=60" 


Other notable event messages: 


When did the cluster recover to the complete state again? (last entry) 


CMReplicationRegistry Finished replication... 
CMBucket event=replicationDone... 

CMMaster replication success... 

CMBucket event=searchableDone... 

CMMaster change bucket success... 


When did the downed peer rejoin the cluster? 


CMMaster scheduled rebalance primaries 
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Module 3 Lab Exercise — Migrate Single-site Cluster to Multisite Cluster 


Description 


In this exercise, you will migrate the single-site indexer cluster you configured in the previous lab exercise 
to a multisite cluster with two sites. The first two indexers, idx1 and idxz2, will be assigned to site1. The 
third indexer, idx3, will be assigned to site2. The original search head, sh1, will be assigned to sitet. 


In order to meet the minimum number of nodes to form a multisite cluster with search affinity, you will add 


an additional indexer idx4 and a search head sh2 for site2. Once the cluster reaches the complete 
state, you will simulate a site failure scenario. 


Steps 
Task 1: Migrate the single-site manager node to the multisite mode. 


1. Open the Misc-Server session and migrate the cmanager instance to support a multisite cluster. 
Use the following options: 


e Manager node site association sitel 


e available sites site1 & site2 

e site replication factor origin:1,total:2 

e site search factor origin:1,total:2 

e replication factor 1 (it was set to 2 in module 2) 
e search_factor 1 (it was set to 2 in module 2) 
e secret idxcluster 


~/cmanager/bin/splunk edit cluster-config -mode manager -multisite true -site sitel 
-available sites site1,site2 -site replication factor origin:1,total:2 


-Site search factor origin:1,total:2 -replication_ factor 1 -search factor 1 -secret 
idxcluster 





~/cmanager/bin/splunk restart 


2. To perform the migration of peers, enable maintenance mode on the manager node. 


~/cmanager/bin/splunk enable maintenance-mode 

Warning: In maintenance mode, the cluster manager will not attempt to 
replace any missing replicated or searchable bucket copies. This mode should 
be enabled only while performing maintenance on peers. Do you want to 


Cen lane? |v) ia] £ 


Maln ec nan e mode See 





NOTE: If you are creating a script, append --answer-yes to bypass the prompt: 


splunk enable maintenance-mode --answer-yes 
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Task 2: Migrate the existing three peer nodes to form the multisite indexer cluster. 


NOTE: Each indexing peer in a production environment must run on a dedicated host. However, to 
simulate a working cluster in this lab environment, each host has been configured to run 
multiple Splunk instances. To accommodate this simulation, each instance has been 
carefully assigned unique port numbers. 


Reference the port matrix below to configure each indexer instance. 


Server Name Splunkd-port Web-port Listening-port Replication-port 


8189 8100 9197 9100 
8289 8200 9297 9200 


8389 8300 9397 9300 
8489 8400 9497 9400 





3. Connect to the IDX-Cluster session and convert idx1 and idx2 to be site1 peer nodes. 


ssh 10.0.x.1 


~/1dx1/bin/splunk edit cluster-config -site sitel 
~/1dx1/bin/splunk restart 
~/1dx2/bin/splunk edit cluster-config -site sitel 


~/1dx2/bin/splunk restart 





4. On the cmanager's Indexer clustering page, confirm that both idx1 and idx2 are members of 
site1. idx3 is not listed yet as it has not been assigned. 


5. Convert idx3 to be a site2 peer node. 


~/1dx3/bin/splunk edit cluster-config -site site2 





~/1dx3/bin/splunk restart 
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6. Launch idx4 and configure it as a site2 peer node. 


~/1dx4/bin/splunk start --accept-license 


~/1dx4/bin/splunk edit licenser-localslave -master_uri https://10.0.x.3:8189 


~/1dx4/bin/splunk enable listen 9497 
~/1dx4/bin/splunk disable webserver 


~/1dx4/bin/splunk edit cluster-config -master_uri https://10.0.x.3:8089 -mode peer 
-site site2 -replication_ port 9400 -secret idxcluster 


~/1dx4/bin/splunk restart 





exit 


7. To perform the deferred fixups and rebalancing, disable maintenance mode on the master node. 


[os_user@ip-10-0-x-3 ~]$ 
~/cmanager/bin/splunk disable maintenance-mode 





8. Confirm that all 4 peer nodes are now listed on the cmanager's Indexer clustering page. 


Task 3: Convert sh1 to be a multisite search head for site1. 


9. Connect to the SH-Cluster session. 
10. Migrate the existing search head shi to multisite mode and set it as the search head for site1. 


[os _user@ip-10-0-x-3 ~]$ 
ssh 10.0.x.2 


[os_user@ip-10-@-x-2 ~]$ 


~/shi/bin/splunk edit cluster-master https://10.0.x.3:8089 -multisite true 
-site sitel 
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Task 4: Configure sh2 as the search head for site2. 


11. Start search head sh2 in multisite mode and configure it as the search head for site2. 


NOTE: Don't forget to use the same indexer cluster secret that the manager node has used. 


~/sh2/bin/splunk start --accept-license 


~/sh2/bin/splunk edit licenser-localslave -master_uri https://10.0.x.3:8189 


~/sh2/bin/splunk edit cluster-config -mode searchhead -master_uri 
https://10.0.x.3:8089 -site site2 -secret idxcluster 


~/sh2/bin/splunk restart 


exit 





Task 5: Check the cluster status with Splunk Web. 


12. Go to cmanager's Splunk Web, https://{Public_DNS}/cmanager and check the cluster status. 
a. Navigate to Settings > Indexer clustering. 


b. Confirm that 4 peers and 3 search heads are shown, and all status indicators are green. 





v All Data is Searchable V Search Factor is Met Y Replication Factor is Met 
4 searchable (0) not searchable 2 searchable (0) not searchable 
Peers Indexes 
Peers (4) Indexes (2) Search Heads (3) 
filter Q 10 per page v 
i Peer Name + Site ^ Fully Searchable + Status > Buckets > ° 
> idx1 site1 v Yes Up 14 
> idx2 site v Yes Up 12 
> idx3 site2 v Yes Up 16 
> idx4 site2 Zv Yes Up 8 
Peers (4) Indexes (2) Search Heads (3) 
filter Q | 10 per page v 
i Search head name + Site + Status + 
> cmanager site1 Zv Up 
> sh1 site1 Zv Up 
> 


sh2 site2 Zv Up 
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Check Your Work 


Task 6: Run a search from sh1 to establish a search result baseline. 


13. Access sh1's Splunk Web by pointing the browser to https: //{Public_DNS}/sh1. 
14. Search the following in the Search app (All time): 


index=_internal host=idx* component=CMSlave Maintenance mode finished | stats 
latest(_ time) as t1 | eval t2 = time() | eval custom_range = "starttimeu=" + t1 | 
eval custom_range = custom_range + " endtimeu=" + t2 | fields custom_range 


index=_internal host=idx* component=CMSlave Maintenance mode finished | stats latest(_time) as t1 | eval t2 = time() | eval custom_range = All time v EJ 
"starttimeu=" + t1 | eval custom_range = custom_range + " endtimeu=" + t2 | fields custom_range 


v 14 events (before 10/14/20 9:20:15.000 PM) No Event Sampling v Job v eo 8 4 ? Smart Mode v 


Events Patterns Statistics (1) Visualization 





20 Per Page v» Format Preview v» 


custom_range + Pa 


starttimeu=1602710385.311 endtimeu=1602710416. 154722 


NOTE: This search example provides a fixed search window that you can use to validate a 
Splunk cluster site recovery test. 


15. Copy the calculated search window value (custom_range) from the result. 
Example above: starttimeu=1602710385.311 endtimeu=1602710416.154722 


16. Run a baseline search that displays the event count per host and its data provider: 


index=_internal host=idx* {copied value of custom_range} 
| stats count by host splunk_server | sort host 


Z 789 events (before 10/14/20 9:20:16.154 PM) No Event Sampling v @ Jobv 2 8 y4 ® Smart Mode v 
Events Patterns Statistics (4) Visualization 

20 Per Page v f Format Preview ¥ 

host + sS splunk_server > S count è z7 
idx1 idx1 187 
idx2 idx2 188 
idx3 idx2 214 
idx4 idx2 200 


NOTE: Your result may vary depending on the state of replication. Run the search again until total 
event count no longer changes. 
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Configuration Troubleshooting Suggestions 


If your configuration is not returning the expected results, troubleshoot by isolating the issue. 


1. Verify the command syntax and spelling on each instance with: splunk btool check --debug 
2. In Cluster Manager's Splunk Web, search for any cluster errors: 


index=_internal sourcetype=splunkd (warn OR error) component=CM* 


3. Check splunkd.1log of each instance for any errors: 
tail -40 ~/<instance_name>/var/log/splunk/splunkd.1log 


4. Compare the output of .conf files with Llab_conf_outputs. txt. 


S lf you are done with the configuration and have about 
—) 15 minutes to spare, try this optional failover test. 


Optional: Simulate Site1 Failover Test Steps 


Task 7: Test the indexer site1 failover scenario. 


NOTE: The indexer cluster peers are running on a single Linux host and the following step is 
performed only to simulate a site failure. 


17. In the IDX-Cluster session, identify the parent splunkd process of idx1 and stop the process. 


ssh 10.0.x.1 


ps -ef | grep "splunkd -p 8189" 


jane MO WO EA apl und jor eilkissS) Tresa E 
jeune aae A Se W 2 O aN IS e a eE I splunkd p olores tart 
[process-runner] 





kill 


18. Identify the parent splunkd process of idx2 and stop the process. 


ps -ef | grep "splunkd -p 8289" 
jane IL le OS AzS sie. tSiedluiolicl =o “S23 5) aS siceuct 
Jee 2Zesie 2Zoaois Uo 2 WOSGWEILS | Siouluielel jeukcl— | Srodltiig el je) ts Ziss oie Ss are 


kill 
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19. Check the Indexer Clustering: Master Node status page and confirm that the search functionality 
has been restored. 


idx1 and idx2 are marked briefly as Shutting Down and the cluster is not searchable. They should 
transition to Stopped and the search capability will recover quickly. 


You should see a green checkmark next to Fully Searchable for idx3 and idx4. 
Due to the replication factors used in this cluster, the site replication factors cannot be met. 


20. Go back to sh1's Splunk Web: https://{Public_DNS}/sh1. 


21. Re-run the exact search in the Search app with the same time range captured in Step 16. 


index=_internal host=idx* {copied value of custom_range} 
| stats count by host splunk_server | sort host 





v 789 events (before 10/14/20 9:20:16.154 PM) No Event Sampling v © Joby 2 8 y4 @ Smart Mode v 
Events Patterns Statistics (4) Visualization 

20 Per Page ¥ £ Format Preview ¥ 

host $ S splunk_server $ rd count è z7 
idx1 idx4 187 
idx2 idx3 188 
idx3 idx3 214 
idx4 idx4 200 


NOTE: The results still contain events from all indexer peers and the counts should be exactly the 
same as before. However, now the data providers are from only site2 peers. This can take 
some time to complete, if you only see one or two indexers, refresh your screen. 


Task 8: Restore the peer processes for site1. 


22. In the IDX-Cluster ssh session, start the Splunk instances for idx1 and idx2. 


~/idx1/bin/splunk start 
~/idx2/bin/splunk start 


exit 





23. Check the cluster status and verify that all peers are up. 
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24. Re-run the exact search again and verify the result: 


index=_internal host=idx* {copied value of custom_range} 
| stats count by host splunk_server | sort host 


v 789 events (before 10/14/20 9:20:16.154 PM) No Event Sampling v © Jobv 


Events Patterns Statistics (4) Visualization 


20 Per Page ¥ f Format Preview v 


host + f 
idx1 
idx2 
idx3 


idx4 


splunk_server > Pd 
idx2 
idx2 
idx2 


idx2 


BS 4 ®@ Smart Mode v 


countè Z 


187 


188 


214 


200 


NOTE: The event count for each host should be unchanged; however, the data providers are again 
from site1 peers (they may or may not be the same original providers). 
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Module 4 Lab Exercise — Configure and Monitor a Cluster Environment 


Description 


In this exercise, you will complete your indexer clustering environment by deploying an add-on to the 
indexing layer and enabling the Monitoring Console (MC) in distributed mode. MC will be enabled on the 
dserver instance. 


The beg_web_idx add-on is one of the three Buttercup Games app packages used throughout this 
course. They have been placed in the /opt/apps/LSD_apps directory on your Misc-Server. 


po | SearchHead_ | Indexer | Forwarder _ | 


bcg_web_idx (indexer add-on) 
bcg_web (app) 
bcg_web_TA (add-on) 





Steps 
Task 1: Stage the bcg_web_idx app and deploy to the indexer cluster. 
1. Confirm the apps exist in the /opt/apps/LSD_apps directory. 


2. Copy the bcg web_idx app from the /opt/apps/LSD_apps directory to the master-apps directory 
of the manager node. 


NOTE: In a production environment, you will probably use the scp command. 


ls /opt/apps/LSD_apps 


DEE Wiss a oe we E oe Wels A 





cp -r /opt/apps/LSD_apps/bcg web idx ~/cmanager/etc/master-apps 


3. Run the find command to check which custom indexes the app uses. 


Examine the content of its indexes. conf file. 


cd ~/cmanager/etc/master-apps/bcg web idx 
find . -name "“indexes.conf" 
./default/indexes.conf 


cat ./default/indexes. conf 


[web] 

coldPath Stop RLON DE, wao Colori 
homePath SSPLUNK DB/web/db 
maxTotalDataSizeMB = 50000 
thawedPath = S$SPLUNK DB/web/thaweddb 
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4. Configure the web index to automatically replicate across the peer nodes. 


Create an indexes.conf file in the app’s local directory with the repFactor attribute set to auto. 


vi ~/cmanager/etc/master-apps/bcg web_idx/local/indexes.conf 
[web] 


COPEI CO EO 





5. From the manager node, run splunk validate cluster-bundle --check-restart to check for 
any errors and determine whether it requires a rolling-restart. 


To check the status, run splunk show cluster-bundle-status 


os_user@ip-10-0-x-3 ~]$ 

~/cmanager/bin/splunk validate cluster-bundle --check-restart 

Validating new bundle and checking if its application results ina restart. 
Please run 'splunk show cluster-bundle-status' to check the status of the 
bundle validation. 


Created new bundle with checksum=019BB4FA06321CFDE415316AC448A222 
~/cmanager/bin/splunk show cluster-bundle-status 


ALG IL CD Cee AS ve eS all = ZCI OE ec E sitel 
Toe a Oa a BA D Qa EA Tae Ss Ooi) AL 16 7 Il 
Poe sound le—7/ A e D hos a T Ea a 2060/4267 1 
leew vedio erte orn e= e a S A a 
este, louincive: a aa on a e ee 
Lese oon ee T ne o irene ee ee ea WS e E a l a O e 72 2 
lec ane k e eee e el eee ae a e e e 
moe woe e e a a lsume e 
SICeEleuUS=Uo 





6. Ifno errors are reported, deploy the bundle to the peer nodes: splunk apply cluster-bundle 


~/cmanager/bin/splunk apply cluster-bundle 
Warning: Under some circumstances, this command will initiate a rolling 
restart of all peers. This depends on the contents of the configuration 


bundle. For details, refer to the documentation. Do you wish to continue? 
[y/n]: y 
Created new bundle with checksum=019BB4FA06321CFDE415316AC448A222 
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7. Check the status of the applied bundle. 


NOTE: Run splunk show cluster-bundle-status until all peer nodes report a status of Up and 
the active_bundle checksum value matches the checksum value from Step 5. 


~/cmanager/bin/splunk show cluster-bundle-status 
mac cerol U Eee Siecle mes (olde 


Bei] | oypiorel vs 

checksum=01 9BB4FA0 6321CFDE415316AC448A222 

cimo stamp Issel Ss (alia ocal time Thu cepe 43 Alle 
Taree t Duno Ie 

checksum=01 9BB4FA0 6321CFDE415316AC448A222 

timestamp=1538090505 (in localtime=Thu Sep 27 23:21: 
laste Weillacle coc lsuuaclile 

checksum=01 9BB4FA0 6321CFDE415316AC448A222 

lestr Velliclercicn evoe Cee 

GiImestanme—l530090505 (ine local time Thu sep 2a: 2: 
IAS OMe a e e e oTa 

Kac eE enee kreo e r re eoa IMO r e re 

checksum=01 9BB4FA0 6321CFDE415316AC448A222 

timestamp=1538090321 (in localtime=Thu Sep 27 23:18: 


1oF/CABD-FB80-4801-A3FB-DCO7585D90DA site2 

activo Joules IL ise 4 ae SZ CEDERA ILS IevNOa A eye, 2 

Ficese ound le— ORI BEAR» GSZ CE DEA lS CA Caoa 

least vedio aree C ovne =o Cisse time a 1S) S Mey Aces, 2.02 
Lasic one CHES Welliclerrmom e e ee 

WESIEE IO a o a e e 

Sie cusS=Uje 





Task 2: Disable indexing on the manager node and the monitoring console instance (dserver). 


8. To forward all index data from cmanager and dserver, create an outputs.conf for cmanager. 


os_user@ip-10-0-x-3 ~]$ 

vi ~/cmanager/etc/system/local/outputs. conf 
[indexAndForward] 

index = false 


[tcpout] 


defaultGroup = default-autolb-group 
forwardedindex.filter.disable = true 
indexAndForward = false 


[tcpout:default-autolb-group] 
server=10.0.x.1:9197,10.0.x.1:9297,10.0.x.1:939/7,10.0.x.1:949/7 





9. Copy outputs.conf to dserver. 
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cp ~/cmanager/etc/system/local/outputs.conf ~/dserver/etc/system/local/outputs.conf 





NOTE: In a production environment, you will probably use the scp command. 


You will defer the steps to disable the indexing on search heads until you have search head 
clustering configured in Module 7. 


10. Restart only cmanager for now (you will restart dserver in the next task.) 


~/cmanager/bin/splunk restart 





Task 3: Enable the Monitoring Console to run in distributed mode on dserver. 


11. To conveniently group and identify the indexer cluster nodes, add a label for the cluster. 


~/cmanager/bin/splunk edit cluster-config -cluster_label idxc-<user> 





12. To enable the distributed search capabilities from the Monitoring Console, configure the master node 
as a search peer of dserver and configure the dserver instance as a cluster search head with its 
search affinity disabled. 


~/dserver/bin/splunk add search-server 10.0.x.3:8089 -remoteUsername admin 
-remotePassword <pw> 


~/dserver/bin/splunk edit cluster-config -mode searchhead -master_uri 
https://10.0.x.3:8089 -site site@ -secret idxcluster 





13. Restart the instance. 


~/dserver/bin/splunk restart 





14. Log into https://{Public_DNS}/dserver. 

15. Navigate to Settings > Monitoring Console. 

16. Click Settings > General Setup on the Monitoring Console menu bar. 
17. Select Distributed under the Mode option to run MC in distributed mode. 


18. Click Continue and you should get a list of remote instances. 


19. Examine the auto-selected Server roles of each instance and adjust the auto-identified roles 
as needed. 


a. Click Edit > Edit Server Roles on dserver. 
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b. Select only the License Master and Search Head roles. 
Verify cmanager has only Cluster Master and Search Head. 
d. Verify that idx1 - idx4 has only Indexer. 


C. 





Setup 


Current topology of your Splunk Enterprise deployment. Learn more E 


Mode 


Standalone 


Distributed 


This instance 


Reset All Settings 


Apply Changes 


i Instance (host) Instance (serverName) Machine Server roles Custom groups Indexer Cluster(s) Search Head Cluster(s) Monitoring State Problems Actions 
dserver dserver ip-10-0-1-3 License Master idxc-os_user V Enabled E Configured Edit ~ 
Search Head 
Remote instances 
5 Instances filter 
Edit Selected Instances ~ 25 Per Page v” 
i Instance (host)? : Instance (serverName)? ; Machine? ; Server roles Custom groups Indexer Cluster(s) Search Head Cluster(s) Monitoring? : State? ; Problems : Actions 
cmaster cmaster ip-10-0-1-3 Cluster Master idxc-os_user Y Enabled E New Edit ~ 
Search Head 
idx1 idx1 ip-10-0-1-1 Indexer idxc-os_user Y Enabled El New Edit ~ 
idx2 idx2 ip-10-0-1-1 Indexer idxc-os_user Y Enabled E New Edit ~ 
idx3 idx3 ip-10-0-1-1 Indexer idxc-os_user ¥ Enabled El New Edit ~ 
idx4 idx4 ip-10-0-1-1 Indexer idxc-os_user Y Enabled E New Edit ~ 








20. Click Apply Changes when you are ready to save the setup. 


If you get an informational message about sharing roles, ignore and click Save. 


21. Continue on to the next step only when you get the Success! dialog box. 


Success! 


Your changes have been applied. 


It may take a few minutes for your instances to be updated. 


If you encounter an error or no such prompt, repeat steps starting on step 14 one more time. 





22. Click Go to Overview. 


The overview page should display the status of 4 indexers, 2 search heads, 1 cluster manager, and 1 
license manager. 
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Check Your Work 
Task 4: Monitor the indexer clustering service activities from Monitoring Console. 


23. From the Overview page, click on the number (1) next to the Cluster Master panel header. 


The Instance page opens with the manager node information. You can further drill down to the 
resource usage page by selecting an option under Views. 


24. To view the clustering service activities, click Indexing > Indexer Clustering > Indexer Clustering: 
Service Activity. 


For a healthy indexer cluster, the Warning and Error Patterns panel should be empty. 


25. To confirm that the web index from bcg_web_idx app has been deployed, go to Indexing > Indexes 
and Volumes > Indexes and Volumes: Deployment. 


26. From the Indexes panel, click web to drill down to the Index Detail: Deployment dashboard. 


While there are no events in the web index yet, it should be present on all 4 indexers (see the Index 
Structure Overview and the Instances panels). 


27. Confirm that forwarding from cmanager and dserver to the indexing layer is working: 


a. Change the selected Index in the Index Detail: Deployment dashboard from web to _internal. 


b. Scroll down to the Event Count by Hosts panel. 


Event Count by Hosts (6) 
Host $ Event Count = 
idx] 
idx3 


idx2 





idx4 60218 
cmanager 40232 


dserver 33191 


c. You can also verify the forwarding by searching from sh1: 
index=_internal sourcetype=splunkd tcpoutputproc host=* | stats count by host 


Both cmanager and dserver should be listed. If not, verify the outputs.conf settings from 
Step 8. 


Configuration Troubleshooting Suggestion 


1. Compare the output of .conf files with Llab_conf_outputs. txt. 
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Module 5 Lab Exercise — Configure a Forwarder 


Description 


In the lab environment, your dserver instance (10.@.x.3:8189) serves multiple server roles. Configure 
dserver to function as your deployment server. 


When you launched your peer nodes in previous lab exercises, you enabled them to receive data from 
forwarders. In this lab exercise, you will configure a forwarder to use the indexer discovery option using 
the deployment server and install the forwarder add-on portion of the three-part Buttercup Games app. 


pod Search Head | Indexer | Forwarder | 


bcg web_idx (indexer add-on) 
bcg web (app) 
bcg web_TA (add-on) 





Steps 
Task 1: On the manager node, enable the indexer discovery option with forwarder site failover. 


1. Open the Misc-Server session and configure the manager to enable indexer discovery. 
ə Toenable, add the indexer_discovery stanza in server. conf 


e Set pass4SymmKey = idxforwarders 
vi ~/cmanager/etc/system/local/server.conf 


nde eree COV Ery 
pass4SymmKey = idxforwarders 





2. Configure forwarder site failover from site1 to site2. 


~/cmanager/bin/splunk edit cluster-config -forwarder_site failover site1:site2 
The cluster-config property has been edited. 

You need to restart the Splunk Server (splunkd) for your changes to take 
CEECEE 





~/cmanager/bin/splunk restart 


Task 2: Configure the deployment server. 


3. Copy the forwarder configuration app uf_base from /opt/apps to the staging directory in dserver. 


cp -r /opt/apps/uf_base ~/dserver/etc/deployment-apps 
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4. Edit the empty ~/dserver/etc/deployment-apps/uf_base/local/outputs. conf file to enable 
indexer discovery, indexer acknowledgment, and volume-based forwarding. 
e The master_uri is the address to your cluster manager node 
e Set the autoLBVolume size to 256KB (262144) 


os_user@ip-10-0-x-3 ~]$ 
vi ~/dserver/etc/deployment-apps/uf_base/local/outputs. conf 
heesoune | 


defaultGroup = default-autolb-group 


eS Soune scl se Ulu ei woejlliey group] 
indexerDiscovery = idxcl 
useACK = true 

autoLBVolume = 262144 


[Ain rete CLLSIGOWS 157 3 LO P<e ll, | 
laser wien = Imes 7 / UW. See’ 
pass4SymmKey = idxforwarders 





5. Create the ~/dserver/etc/deployment-apps/uf_base/local/props.conf file to enable the 
forwarder event breaking option for all single-line events. 


vi ~/dserver/etc/deployment-apps/uf_base/local/props. conf 
[default] 
J WAEEINTID TE IS ReNSIE EINEM JLy|ty =e eh 





6. Create the ~/dserver/etc/deployment-apps/uf_base/local/server.conf file to configure the 
forwarder to send data to all peers in site1. 


vi ~/dserver/etc/deployment-apps/uf_base/local/server.conf 
PENGEN 
Site = sitel 





7. Copy the bcg _web_TA app from the /opt/apps/LSD_apps directory to the deployment server 
(dserver). 


cp -r /opt/apps/LSD_apps/bcg web TA ~/dserver/etc/deployment-apps 





NOTE: In a production environment, you will probably use the scp command. 
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8. Configure the eng-uf server class so it deploys to both the uf_base and bcg_web_TA add-ons to 
forwarders. Create the server class in ~/dserver/etc/system/local. 


vi ~/dserver/etc/system/local/serverclass. conf 
[sewer lass 2 ein, Une | 
goer Wabi A106 == i005 Oren s 


[Seve lhass Seic, Wie gels Ue lehee| 


restartsSplunkWeb = 
restartSplunkd = 1 
stateOnClient = enabled 


Poer er CaS seid, le Bays OCENE OEA] 
restartSplunkWeb 

PESE EE S[oUL Wael 

stateOnClient 





9. Reload the server class. 


~/dserver/bin/splunk reload deploy-server 
Reloading serverclass(es). 





Task 3: Enable the deployment client setting on the forwarder. 


10. Start the forwarder with the auto-ports option and configure it as a deployment client. 


In your lab environment, the forwarder instance is installed in the fwdr directory of Misc-server. 


[os _user@ip-10-0-x-3 ~]$ 
~/fwdr/bin/splunk start --accept-license --answer-yes --auto-ports --no-prompt 


~/fwdr/bin/splunk set deploy-poll 10.0.x.3:8189 
Splunk username: admin 


Password: 





Cone eo ea e sien’ a e 


Task 4: Update the instance server role in Monitoring Console. 


11. Log into https://{Public_DNS}/dserver. 
12. Navigate to Settings > Monitoring Console. 


13. Click Settings > General Setup on the Monitoring Console menu bar. 
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14. -Examine the auto-selected Server roles for dserver and adjust the auto-identified roles as needed. 


a. Click Edit > Edit Server Roles on dserver. 

b. Add Deployment Server to the selected server roles. 
c. Click Save > Done. 

d. Click Apply Changes > Save. 


15. When you get the Success! dialog box, click Go to Overview. 


The overview page should now include 1 deployment server. Refresh the page until the client count in 
the deployment server panel changes to 1. 


Check Your Work 
Task 5: Verify the forwarder app deployment. 


16. Log into https: //{Public_DNS}/sh1. 


17. Search the last 15 minutes of internal logs to confirm the indexer discovery activities on the forwarder: 
index=_internal host=uforwarder TcpOutputProc | timechart values(event_message) 
as messages | search messages=* 


NOTE: Be patient. Take a quick break. It will take about 3 - 5 minutes to complete the deployment. 


The stats should list only the peers from site1. 
index=_internal host=uforwarder TcpOutputProc | stats count by splunk_server 
18. Search the last 15 minutes of internal metrics to verify input data is flowing to the web index: 
index=_internal sourcetype=splunkd component=Metrics series=web | timechart 


Span=1m max(kb) by series 


19. To confirm, search the web index: 


index=web (Last 24 hours) 


After you confirm that data is flowing to the web index, test the forwarder site failover. 
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Task 6: Test the forwarder site failover scenario. 
20. In the IDX-Cluster session, identify the parent splunkd process of idx1 and stop the process. 


ssh 10.0.x.1 


ps -ef | grep "splunkd -p 8189" 


jane TOn eee 0st Asm ulin Sel Oks mere cir rE 
Jens Zoe! Z2ooce Wo 2 WU SOE IS | soulimeleel joc | “fsreubitiig el je) soils Sh ieeS eciiaie 
[process-runner] 


kill 





21. Identify the parent splunkd process of idx2 and stop the process. 


ps -ef | grep "splunkd -p 8289" 
jane kG) 2 WOsA2rzss Solltiakel =o S238) wSsiceuac 
Jeune Zoo o T W 2 Wess ils | syollunel<el jonticl |) iollitlig cl je) Ss o Oro seca 


[process-runner] 


kill 





22. Go to the Monitoring Console Overview page on https://{Public_DNS}/dserver. 


NOTE: The indexers panel should report that two instances are not reachable. 


23. Search the last 15 minutes of internal metrics to verify that the forwarder failover is working: 


index=_internal host=uf* component=Metrics group=tcpout_connections | timechart 
Span=im sum(kb) by destPort 


NOTE: The time chart should display the destination switchover. The failover can take up to 5 
minutes. You may need to adjust the search window and timechart span. 


For the visualization, change the Scale option in Format to Log on the Y-Axis tab. 
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Events Patterns Statistics (16) Visualization 


al Column Chart #/ Format ge Trellis 
100,000 


10,000 


| I I I | 


11:56 PM 11:58 PM 12:00 AM 12:02 AM 12:04 AM 12:06 AM 12:08 AM 12:10 AM 





24. To restore the service, start Splunk for idx1 and idx2 in the IDX-Cluster ssh session. 


~/idx1/bin/splunk start 
~/idx2/bin/splunk start 


exit 





25. Confirm that the warning message is cleared from the Monitoring Console page. 
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Configuration Troubleshooting Suggestions 


If your configuration is not producing the expected results, troubleshoot by isolating the issue. 


1. Verify that the forwarder has downloaded the uf_base app and restarted. 


~]$ 
ls ~/fwdr/etc/apps 


Degiweb AWS. mtr o PECE TONE Ner iror elelom eciedecl Stecicel  sjolulail< [gcic Sligo. c 





SplunkUniversalForwarder uf_base 


2. On the cmanager, search the internal index to check if the forwarder has contacted the 
manager node. 


index="_internal" component=CMIndexerDiscovery host=cmanager 


You should have an event indicating a new forwarder has contacted the cluster manager: 
CMindexerDiscovery — Registering new forwarder <some GUID> (total; IJ. 
Heartbeat assigned for next check: 30 seconds 


If you see such an event, then go to Troubleshooting Step 3. 


If you get no result, then stop and check the master_uri value specified in 
~/dserver/etc/deployment-apps/uf_base/local/outputs.conf. 


3. Check the forwarder splunkd.1og for any failed heartbeat messages or any other clues: 


tail -100 ~/fwdr/var/log/splunk/splunkd.1log 
tail ~/fwdr/var/log/splunk/splunkd.log | egrep 'HttpPubSubConnection' 
tail -f ~/fwdr/var/log/splunk/splunkd.log | egrep 'TcpOutProc' 


A failed heartbeat message indicates an issue with the value of pass4SymmKey. Make sure the same 
value is used for the cluster manager and the forwarder. If you are not sure: 


a. Set it again in clear-text on both cmanager (Configuration Step 1) and fwdr 
(Configuration Step 4) 


b. Restart cmanager and then the fwdr. 
A failed to extract FwdTarget message indicates misconfigured listening ports on the indexer peers. 


4. Compare the output of .conf files with lab_conf_outputs.txt. 
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Module 6 Lab Exercise — Enable Search Head Cluster 


Description 


Currently, you have two dedicated search heads — one for each site -- independently searching the 
indexer cluster sites. 


In this lab exercise, you will add the existing site2 search head to a new 3-member search head cluster. 
Keep the site1 search head as stand-alone for comparison. To complete the search head cluster, you 
will launch two more search head instances and associate them with site2. You will also integrate the 
new search head cluster to your existing indexer cluster. 


After the search head cluster is up, you will run a quick test to validate its functionality. 
Steps 


Task 1: Add two more search heads to site2. 


NOTE: Each member in a search head cluster in a production environment must run on a dedicated 
host. However, to simulate a working search head cluster in this lab environment, a single 
host is configured to run multiple Splunk instances. 


To accommodate this simulation, each instance has been carefully assigned unique port 
numbers. Reference this port matrix to configure each search head instance 


ServerName | Splunkd-port Web-port Replication-port | 
sh1 9109 


8289 8200 9200 


8389 8300 9300 
8489 8400 9400 


1. In the SH-Cluster session, bring up sh3 and sh4 as search heads for site2. 





a. Configure search heads as license peers to dserver. 


b. Integrate search heads with the site2 indexer cluster. 
Hint: Don't forget to use the same secret that the manager node has used. 


[os _user@ip-10-0-x-3 ~]$ 

ssh 10.0.x.2 

~/sh3/bin/splunk start --accept-license 

~/sh3/bin/splunk edit licenser-localslave -master_uri https://10.0.x.3:8189 


~/sh3/bin/splunk edit cluster-config -mode searchhead -master_uri 
https://10.0.x.3:8089 -site site2 -secret idxcluster 


~/sh3/bin/splunk restart 
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~/sh4/bin/splunk start --accept-license 


~/sh4/bin/splunk edit licenser-localslave -master_uri https://10.0.x.3:8189 


~/sh4/bin/splunk edit cluster-config -mode searchhead -master_uri 
https://10.0.x.3:8089 -site site2 -secret idxcluster 





~/sh4/bin/splunk restart 


2. On cmanager, check the Cluster Manager Node status page and confirm that site2 now has three 
search heads. 





Peers (4) Indexes (4) Search Heads (6) 
filter Q 10 per page ¥ 

i Search head name $ Site ~ Status $ 
v Up 
v Up 
v Up 

> cmanager site1 v Up 

> shi site1 v Up 

> dserver siteO v Up 


Task 2: Enable a search head cluster with site2 search heads. 


3. Inthe SH-Cluster session, initialize sh2 to be a member of the search head cluster. 


Use splunk init shcluster-config with: 


ə mgmt_uri https://10.0.x.2:8289 
e replication_port 9200 
e secret shcluster 


Repeat the process to initialize sh3 and sh4 to be members of the same search head cluster. 


NOTE: To get more CLI help, runsplunk help init shcluster-config 


You are allowed to initialize sh2 without re-installing Splunk because no knowledge objects 
and artifacts were created on this instance. 


Reference the port matrix to configure each search head instance. If you get the "This 
command needs splunkd to be up, and splunkd is down" message, it probably means there 
is a typo or syntax error in the command. 
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[os _user@ip-10-0-x-2 ~]$ 
~/sh2/bin/splunk init shcluster-config -mgmt_uri https://10.0.x.2:8289 
-replication port 9200 -secret shcluster 


~/sh3/bin/splunk init shcluster-config -mgmt_uri https://10.0.x.2:8389 
-replication_ port 9300 -secret shcluster 


~/sh4/bin/splunk init shcluster-config -mgmt_uri https://10.0.x.2:8489 
-replication_ port 9400 -secret shcluster 


~/sh2/bin/splunk restart 


~/sh3/bin/splunk restart 





~/sh4/bin/splunk restart 


4. Bootstrap sh2 to be the initial captain with the splunk bootstrap shcluster-captain command 
and add members one by one with the splunk add shcluster-member command. 


~/sh2/bin/splunk bootstrap shcluster-captain -servers list https://10.0.x.2:8289 


~/sh2/bin/splunk add shcluster-member -new_member_uri https://10.0.x.2:8389 





~/sh2/bin/splunk add shcluster-member -new_member_uri https://10.0.x.2:8489 


NOTE: You will learn more about adding SHC members in Module 7. 
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5. Check the running state of the search head cluster from any member. 


~/sh2/bin/splunk show shcluster-status 
Captain: 


Kreme Weystceiain = Ah 
olo ted vecormeia — Neel er jo. Aheoseeis. Cn? 
TCL RC DEA ah Za Byte 40S DS 2D) C= Diy os SIZ) DIDS 2 
Lig iicde IL iaexcl ile = 1 
label : sh2 
mome eur e e e a OO a S, 
Min peers JOlnec lag = I 


no Manae o a e a Fv 
SEn e ea itikew & 
MEMOCES: 
sh4 
label : sh4 
las eeno iole ee e on E IWecl Occ bo a T O 2019 
mone “eds a e epo 7 OO 23 te us 
Mente Wie Silas 2 a a a a 
Sieve = Ue 





6. To easily group and identify the search head cluster members, add the label for the cluster. 


~/sh2/bin/splunk edit shcluster-config -shcluster_label shc-<user> 





When the search head cluster is up and running for the first time, it automatically disables Monitoring 
Console from all members. For this to fully take effect, you need to restart all members. 


7. Torestart the search head cluster, execute the rolling-restart from the captain's command line. 


[os _user@ip-10-0-x-2 ~]$ 
~/sh2/bin/splunk rolling-restart shcluster-members 


~/sh2/bin/splunk rolling-restart shcluster-members -status 1 
Rolling restart Success : l 

Message : Rolling Restart of all the search head cluster members has been 
kicked off. It might take some time for completion. After restart the 
information will be logged at audit log, Meanwhile you can check the 


progress of this transaction using 


ECs oulbig<=) rolling restart shecluscer- members -otarus IE 
exit 
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Check Your Work 


Task 3: Verify that your search head cluster is functioning properly. 


8. Log into your sh3: https://{Public_DNS}/sh3. 
Navigate to Settings > Search head clustering. 
Identify the current captain and wait until all members are Up. 
10. In the Search app, search over the Last 15 minutes: 
index=_internal sourcetype=splunkd error 
11. Save the search as a scheduled report. 
e Title: My Cluster Errors - last 15 minutes 
e Time Range Picker No 
12. Click Schedule and set the following: 
e Schedule Report v 
Schedule Run on Cron Schedule 
e Cron Expression TAO) EE eg 
e Time Range Last 15 minutes 
e Schedule Priority Default 
e Schedule Window No window 
Do not enable any actions and Save. 
13. Open another browser tab and go to https://{Public_DNS}/sh4. 
a. Go to the Search & Reporting app and click Reports. 
b. Confirm that the My Cluster Errors - last 15 minutes scheduled report from sh3 has been 
replicated on this member. 
c. Runthe My Cluster Errors - last 15 minutes report. 
NOTE: Ignore the message, "There are no results because the first scheduled run of the report has 
not completea" for now. 
d. Navigate to Settings > Users. 
e. Click New and create an account for emaxwel11 with the following field values: 
e Username: emaxwell 
e Assign to roles: power and user 
e Password: (for simplicity, use the same password as admin) 
e Confirm password: (for simplicity, use the same password as admin) 
e Require password change on first login: Uncheck 
14. Open another browser tab and log into https: //{Public_DNS}/sh2 as emaxwel1l. 
If you are able to log in as emaxwel11, log out. 
15. Log back into https://{Public_DNS}/sh2 as admin. 
16. Confirm that the My Cluster Errors - last 15 minutes scheduled report from sh3 has been 


replicated on this member. 
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17. 
18. 


19. 
20. 


Click Activity > Jobs and confirm that the search job from the scheduled report is shown. 
Click the Job > Inspect Job link associated with the matching search. 


The SID includes the GUID of the member who ran the search. Click Search job properties to 
expand the properties. Scroll to the bottom of the page to find the searchProviders info. From the 
searchProviders value, you can also deduce which member ran the job. 


Go to https://{Public_DNS}/sh1 


Verify that there is no such report, no job artifacts, nor the user emaxwel11 on this search head. 


Troubleshooting Suggestions 


If your configuration is not returning the expected results, troubleshoot by isolating the issue. 


1. 
2. 


Verify the command syntax and spelling on each instance with: splunk btool check --debug 
From the would-be captain, search (Last 60 minutes) for the captain election messages: 
index=_internal sourcetype=splunkd component= SHCRaftConsensus | reverse 
Check splunkd. log of each instance for any errors: 

tail -40 ~/<instance_name>/var/log/splunk/splunkd. 1log 


Compare the output of . conf files with lab_conf_outputs.txt. 
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Module 7 Lab Exercise — Deploy an App to SHC 


Description 


In this exercise, you will complete your Splunk clustering environment and perform basic administration 
tasks such as deploying an app and monitoring the search head cluster activities. 


You will deploy the final part of the three-part Buttercup Games Web app into the SHC with deployer. In 
this lab environment, you will configure dserver (10.@.x.3:8189) to function as your deployer instance. 


po SearchHead | Indexer | Forwarder 
| 


beg_web_idx (indexer add-on) ee ee 
beg web (app) a a ee 
beg_web_TA (add-on) Ooo S ee ee ëf 


Steps 





Task 1: Configure the SHC members and the deployer. 
1. To enable the deployer component on dserver, add the search head cluster's pass4SymmKey in 
server .conf. 
vi ~/dserver/etc/system/local/server. conf 


[shclustering] 
pass4SymmkKey = shcluster 


~/dserver/bin/splunk restart 
ssh 10.0.x.2 





2. Add the deployer's address to the existing search head cluster members. 


Defer splunk restart for now. You will use the web UI after all members are edited. 


~/sh2/bin/splunk edit shcluster-config -conf_deploy fetch_url https://10.0.x.3:8189 


~/sh3/bin/splunk edit shcluster-config -conf_deploy fetch_url https://10.0.x.3:8189 


~/sh4/bin/splunk edit shcluster-config -conf_deploy_ fetch_url https://10.0.x.3:8189 


exit 
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3. Log into any SHC member and navigate to Settings > Search head clustering. 
4. Click Begin Rolling Restart > Restart. 


You may continue on to Task 2 while SHC processes the rolling-restart. 


Task 2: Stage and distribute apps to search head cluster members. 


5. Copy an existing app bcg web in the /opt/apps/LSD_apps directory to the deployer's 
shcluster directory. 


NOTE: In a production environment, you will probably use the scp command. 


[os_user@ip-10-@-x-3 ~]$ 


cp -r /opt/apps/LSD_apps/bcg web ~/dserver/etc/shcluster/apps 





6. Create a new app called shc_base in the deployer's shcluster directory. 


NOTE: This app disables indexing on the search head members. In outputs.conf, be sure to use 
the correct IP addresses and ports for your indexer peer nodes. 


mkdir -p ~/dserver/etc/shcluster/apps/shc_base/{default,metadata} 
touch ~/dserver/etc/shcluster/apps/shc_base/metadata/local.meta 


vi ~/dserver/etc/shcluster/apps/shc_base/default/app. conf 
[ui] 
LS WaSiole = © 


[package] 
TOR Slice Sesic 
ENE HOM UScacss = 0 


vi ~/dserver/etc/shcluster/apps/shc_base/default/outputs. conf 
[indexAndForward] 


index = false 


eelorerbhe | 

defaultGroup = default-autolb-group 
forwardedindex. iL Licene sicllsclolkSs - Ctre 
indexAndForward = false 


Peep oul: de tault—autolb—or ou) 
Server Ws Oe elle ole 7.10.0 ee Lee 7 Oe a BS LOO aka Le a) 





© 2021 Splunk Inc. All rights reserved. Splunk Cluster Administration - Lab Exercises February 25, 2022 46 


splunk > 





7. To manually control the deployment of your staged apps to the search head members, run splunk 
apply shcluster-bundle with the -action parameter from the deployer. 


You may use any member of the search head cluster as the target. 


~/dserver/bin/splunk apply shcluster-bundle -action stage --answer-yes 





NOTE: Ignore the message Bundle has been pushed successfully to all the cluster 
members in this phase. No actual apps have been sent yet. This step only checks the validity 
of your app bundles. 


~/dserver/bin/splunk apply shcluster-bundle -action send -target 
https://10.0.x.2:8289 --answer-yes 





NOTE: The message Bundle has been pushed successfully to all the cluster members 
indicates a successful execution. To confirm, search the internal logs. 


8. Log into https://{Public_DNS}/dserver and search the following for confirmations (last 15 
minutes): 


index=_internal confdeployment data.task=createDeployableApps | table host, 
data.task, log level, data.source_area, data.staging area (This confirms the staging.) 


index=_internal confdeployment data.task=sendDeployableApps 
| table host, data.target_label, data.target_uri, data.status (This confirms the 
sending.) 


Check Your Work 
Task 3: Verify the app deployment. 


9. Log into https://{Public_DNS}/sh2. 
The BCG Web app should appear in the list of apps. 


10. To verify the successful app deployment from the deployer, repeat the steps on all remaining search 
head members. 
e https://{Public_DNS}/sh3 
e https://{Public_DNS}/sh4 
11. On any search head member (e.g. sh4), test the following: 
a. Go to the BCG Web app. 
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b. Confirm the Web Store Status dashboard panels are populating with results. 


Task 4: Complete the Monitoring Console setup on dserver. 


12 


. Add sh2, sh3, and sh4 as search peers of dserver. 


~/dserver/bin/splunk add search-server 10.0.x.2:8289 -remoteUsername admin 


remotePassword <pw> 


~/dserver/bin/splunk add search-server 10.0.x.2:8389 -remoteUsername admin 


remotePassword <pw> 


~/dserver/bin/splunk add search-server 10.0.x.2:8489 -remoteUsername admin 
remotePassword <pw> 


13 


14. 


15. 


16. 


17. 





. Log into https://{Public_DNS}/dserver. 
Navigate to Settings > Monitoring Console. 
Click Settings > General Setup on the MC menu bar. 


Examine the Server roles of each instance again and adjust the auto-identified roles if required. 


a. Click Edit > Edit Server Roles on dserver. 
b. Add the SHC Deployer role. 
c. Click Save > Done. 


Examine the Server roles of each SHC member instance and adjust the auto-identified roles. 


a. Select the check box next to sh2, sh3 and sh4. 

b. Click Edit Selected Instances > Set Server Roles. 
c. Select only the Search Head and KV Store roles. 
d. Click Save > Done. 
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Setup 


Current topology of your Splunk Enterprise deployment. Learn more E 
Mode Standalone Distributed Reset All Settings Apply Changes 


This instance 


i Instance (host) Instance (serverName) Machine Server roles Custom groups Indexer Cluster(s) Search Head Cluster(s) Monitoring State Problems Actions 
dserver dserver ip-10-0-1-3 Search Head idxc-onez vY Enabled E Configured Edit ~ 
License Master 
Deployment Server 
SHC Deployer 


Remote instances 


8 Instances filter 


Edit Selected Instances ~ 25 Per Page 7 








i Instance (host)? : Instance (serverName) ? + Machine? ; Server roles Custom groups Indexer Cluster(s) Search Head Cluster(s Monitoring? : State? ; Problems ; Actions 
cmaster cmaster ip-10-0-1-3 Cluster Master idxc-onez VY Enabled E Configured Edit ~ 
Search Head 
idx1 idx1 ip-10-0-1-1 Indexer idxc-onez V Enabled E Configured Edit ~ 
idx2 idx2 ip-10-0-1-1 Indexer idxc-onez ¥ Enabled E Configured Edit ~ 
idx3 idx3 ip-10-0-1-1 Indexer idxc-onez ¥ Enabled E Configured Edit ~ 
idx4 idx4 ip-10-0-1-1 Indexer idxc-onez VY Enabled E Configured Edit ~ 
y sh2 sh2 ip-10-0-1-2 Search Head idxc-onez shc-onez ¥ Enabled E New Edit ~ 
KV Store 
Y sh3 sh3 ip-10-0-1-2 Search Head idxc-onez shc-onez ¥ Enabled E New Edit ~ 
KV Store 
xá sh4 sh4 ip-10-0-1-2 Search Head idxc-onez shc-onez ¥ Enabled E New Edit + 
KV Store 





18. Click Apply Changes when you are ready to save the setup. 


If you get an informational message about sharing roles, ignore and click Save. 
19. Continue on to the next step only when you get the Success! dialog box. 
Success! 


Your changes have been applied. 


It may take a few minutes for your instances to be updated. 


20. Click Go to Overview. 





NOTE: The overview page should now display the status of 4 indexers, 5 search heads (because 
shi is not configured), 1 cluster master, 1 license master, and 1 deployment server. 
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Task 5: Review the search head cluster dashboards. 


21. Click Search > Search Head Clustering > Search Head Clustering: Status and Configuration on 
the MC menu bar. 


a. Confirm that the Health Check panel indicates no issue. 
b. Check the Status panel for the current captain and its term. 


22. Navigate to the Search Head Clustering: App Deployment dashboard and confirm that the status 
of each app is Synchronized. 


Snapshots 
Apps Status 
2 apps 
App + Status $ 
BCG Web Synchronized 


shc_base Synchronized 


Troubleshooting Suggestions 


If your configuration is not returning the expected results, troubleshoot by isolating the issue. 


1. Verify the command syntax and spelling on each instance with: splunk btool check --debug 


2. To check the deployer deployment status, search the last 60-minutes of internal events in dserver. 
index=_internal component=ConfDeployment data.task=sendDeployableApps | table 
data.target_ label data.status 

3. Check splunkd.1log of each instance for any errors: 
tail -40 ~/<instance_name>/var/log/splunk/splunkd.1log 


4. Compare the output of .conf files with Llab_conf_outputs. txt. 
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Module 8 Lab Exercise — Add a KV Store Collection 


Description 

The bcg_web app you have deployed to your search head cluster in the previous module is shipped with 
collection configurations and a CSV file you can use to populate the initial KV store collection. However, 
the collection will not work with the indexer cluster because the app is configured for a non-clustered 
deployment. 


In this lab exercise, you will manage a KV store lookup to work within a clustered search head 
environment. 


Steps 


Task 1: Identify the current SHC captain and KV store captain. 


1. Inthe SH-Cluster session, identify the current SHC captain. 


ssh 10.0.x.2 


~/sh2/bin/splunk show shcluster-status 
Captain: 


dynamic WeysicelLige = AL 
SlGCrs>e Caceena § (ue Cee Is 2Zoe0l 20° 2020 


TG 2 AG DEAZ Eh Be — 41S DS 2D) C= Dien G4 SIL) Z DIS SV 
Tarea JMS Ie ikele, 2k 
label : sh2 
mgmt uri : https://10.0.x.2:8289 
na aee emne e a = 
Po Mrge e ear ce A: 
Se Ea a: 





NOTE: In this example, sh2 is identified as the SHC captain. 
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2. Runthe show kvstore-status command to identify the current KV store captain (primary). 


~/sh2/bin/splunk show kvstore-status 


This member: 
dare 
TS S Sc 
disabled 
TO 
oplogEndTimestamp 
oplogEndTimestampSec 
oplogStartTimestamp 
oplogStartTimestampSsec 
POTE 
replicaset 
replicationStatus 
Stance lone 
SECAR S 


Enabled KV store members: 


KV store members: 
10.0.x.2:8491 
CONE E Oro ON 
electionDate 
electr ronDatessc 
hostAndPort 
optimeDate 
optimeDateSec 


replicationStatus 


uptime 


NOTE: 


aoreet oone ari STe eee Oe) 
LOOT VES EEA 

O 

97 7E4E87-02A4-4AF0-BCBB-4BB088470525 
MONT OC CEE ZS She Wve ZleZ(0) 
LOSS Se Las 

Te VOC elo Cue eal Ge OO) 
PO O40 a6 

SS 

STOW MNaN ies: 

Non-captain KV store member 
O 

ready 


ik 

FueVOC Else o rO Eoo 
E E Se 

10.0.x.2:8491 

dibreeeGohes Isr A a e A 
BSOD hD 

KV store captain 

2200 





In this example, sh4 (10.0.x.2:8491) is identified as the KV store captain. 


Task 2: Verify the state of the KV store service from Monitoring Console. 


3. 


4. 


NOTE: 


In dserver's Monitoring Console, navigate to Search > KV Store > KV Store: Deployment. 
In the Warning and Error Patterns panel, change the Time Range to Last 15 minutes. 


The panel should not have any reported issues. 


5. Review the KV Store Status panel and identify the primary instance in the Replication Role column. 
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Snapshots 


KV Store Status 


Total Active 
Instance Physical Memory Mapped Memory Page Faults per Queued Connections Lock Last Flush Network Uptime Replication 
$ Usage (MB) $ Usage (MB) + Operation + < z (%) > (ms) $ Traffic (MB) + (hours) + Role $ 


sh2 0.00 0 17 5 10.34 0.64 Secondary 
sh3 0.00 ts) 18 6 12.87 0.65 Secondary 


sh4 0.00 0 22 2 14.10 0.64 Primary 


Click instance name for more details. Total queued is operations (readers and writers) waiting for a read or write lock to be cleared. 


6. Log into the primary instance's Splunk Web. (sh4 in this example) 
7. Navigate to Settings > Lookups > Lookup definitions. 
8. Verify the product_lookup KV store is configured under the bcg_web app. 
NOTE: The configured KV store supports the following lookup fields: 
productId, product_name, categoryId, price, sale price, Code 
9. Navigate to the BCG Web app > Search, then run the following search: 


| inputlookup product_lookup 


NOTE: This search returns the No results found message because the KV store collection is 
currently empty. 


10. Run a search to populate the KV store with the CSV file contents in the lookups directory: 


| inputlookup products.csv | outputlookup product_lookup 


NOTE: Note the processed count on the Statistics tab. It should be 15. Click the Job menu to 
display a message indicating that the records were written to product_collection and 
there were warnings. 


11. In dserver's Monitoring Console, navigate to Search > KV Store > KV Store: Instance. 


NOTE: The number of objects in the product_collection shows 14, regardless of which SHC 
member instance you select. Why? 
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12. To investigate, go to Activity > Jobs page on any SHC member (sh4 for example) and select the 
Inspect Job option associated with the search from Step 10. 


NOTE: You should have messages such as: 


info : Results written to collection ‘product_collectton'. 
info : Successfully read Lookup file '.../bcg_web/Lookups/products.csv'. 


warn : There were warnings when executing this outputlookup. See 
search.log for more information. 


13. Click search.log and find a log entry WARN KVStoreLookup. 
NOTE: You should have messages such as: 


WARN KVStoreLookup - Skipping row due to bad value '{[@]='NA'}' for field 
‘price’ (expected type: 1) 


INFO outputcsv - 14 events written to product collection 


These are the process artifacts from KV store data type enforcement. 


14. Run the search from Step 10 again and confirm that you get 14 results. 
15. Run a search to verify the lookup knowledge enhancement (All time): 


index=web sourcetype=access_ combined action=purchase | lookup product_lookup 
productId | stats sum(price) by product_name 
a. Open the job inspector and write down the search duration. 


b. Scan the Execution costs histogram and record the values of the command.stats and 
dispatch.stream.remote components. 


Example: This search has completed and has returned 14 results by scanning 26,995 
events in @.782 seconds 


Duration Component Invocations Input count Output count 
0.03 command. stats 28 26,996 14 
0.57 dispatch.stream. remote 22 - 12,870,775 


c. Expand the Search job properties and click the remote log from idx3. 


d. Check if the remote search performed any lookups by searching for product_lookup. 


NOTE: No product_lookup entry exists in any of the idx#.1og files. 
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Task 3: Enable the KV store collection replication. 


16. In the Misc-Server session, edit the bcg_ web app's collections. conf file in the deployer's 
shcluster directory to enable the collection replication. 


Apply the bundle after the edit. 


vi ~/dserver/etc/shcluster/apps/bcg web/default/collections. conf 
Product E o M eeren 

SNERO CEI ee = Ene 

field.price = number 

Eele Sele joewes = ious 

1 at) oJ i Kof- ho oo gC) 


~/dserver/bin/splunk apply shcluster-bundle -target https://10.0.x.2:8289 





NOTE: Updating collections.conf does not require a search head cluster rolling-restart. 


17. Run the same search in BCG Web to verify the lookup replication (All time): 


index=web sourcetype=access_ combined action=purchase | lookup product_lookup 
productId | stats sum(price) by product_name 
a. Open the job inspector and compare the execution costs to the values you recorded earlier. 


Scan the Execution costs histogram and compare the command.stats and 
dispatch.stream.remote values you recorded earlier. 


c. Expand the Search job properties and click the remote log from idx3. 


Check if the remote search performed any lookups by searching for product_lookup. 
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Example: 


NOTE: 





This search has completed and has returned 14 results by scanning 27,012 
events tn @.511 seconds 


Duration Component Invocations Input count Output count 
0.01 command .stats 18 97 14 
0.24 dispatch.stream. remote 11 = 97,155 


product_lookup log events can be found in all logs now. 


SearchParser - PARSING: Litsearch (action=purchase index=web 
sourcetype=access combined) | Lookup product Lookup productId | addinfo 
type=count Label=prereport_events track_fieldmeta_events=true | fields 
keepcolorder=t "prestats_reserved_*" "price" "product_name" "psrsvd_*" | 
prestats sum(price) by product_name 


m LL 


The peer nodes were able to filter out the events based on the lookup and send much less 
data to the search head. With the larger dataset, you may see more search performance 
improvement. 


Troubleshooting Suggestions 


1. Ifyou make a mistake and you want to clean the KV store collection and start over, you can run the 
clean kvstore command from the KV store captain's terminal. 


For example, if you have determined that the current KV store captain is on sh4 by running the show 
kvstore-status, you can run the following: 


~/sh4/bin/splunk clean kvstore -app bcg web 


This action will permanently erase KVStore data. 


Are you sure you want to continue [y/n]? y 





2. To confirm the cleaning, search: | inputlookup product_lookup 


NOTE: 


It should return No results found. And also, the number of objects in the MC > Search 
> KV Store > KV Store: Instance page should show @ in all instances. 
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Module 9 Lab Exercise — Migrate the Indexer Cluster to use SmartStore 


Description 

In this scenario, your indexer cluster has been in production for some time and the storage requirement 
has outgrown the compute resources. Unfortunately, your environment doesn't have any room to scale 
out the peer nodes. However, you already have a S3-compliant storage with plenty of storage capacity. 

In order to meet the growing storage requirements, you will migrate your multisite indexer cluster to utilize 
SmartStore to increase the storage capacity without adding more peer nodes. You will use the single 
volume storage option in this exercise. 


Your instructor will provide the following remote storage service access information: 


e path =<s3_bucket_ns> 

e remote.s3.access key =<access key> 

e remote.s3.secret_key = <secret_key> 

e remote.s3.endpoint = <aws_region_uri> 


NOTE: Copy the above attributes to a text editor and replace/add your specific values. 


WARNING: Migrating indexes to SmartStore is a one-way option and cannot be reverted. 
Steps 


Task 1: Verify the SmartStore connectivity from a test instance. 


1. Inthe SH-Cluster session, create a test file. 


ssh 10.0.x.2 





echo “Hello World“ > test99.txt Replace with your student ID 


NOTE: To verify the S3 connectivity and SmartStore functionalities, you are going to start a 
standalone test instance. 


2. Start a test instance sh5. 


~/sh5/bin/splunk start --accept-license 





3. From ~/sh5/bin, verify the SmartStore connectivity using the provided remote storage credentials. 
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e Tolist the content of a s3 bucket, run: 


./splunk cmd splunkd rfs -- --access-key <access key> --secret-key 
<secret_key> --endpoint <aws_region_uri> ls --starts-with <s3_bucket_ns> 


e To copy a file to a s3 bucket, run: 


./splunk cmd splunkd rfs -- --access-key <access key> --secret-key 
<secret_key> --endpoint <aws_region_uri> putF <local_ file> <s3_directory> 


NOTE: The <s3_directory> is defined with <s3_bucket_ns> and <directory>. To test the 
connectivity, this lab environment will share a test directory called all. 


The <s3_bucket_ns> in the following example is s3://1sd1234 and the resulting 
<s3_directory> is: s3://1sd1234/all. 


./splunk cmd splunkd rfs -- --access-key --secret-key 
--endpoint 
ls --starts-with 


./splunk cmd splunkd rfs -- --access-key --secret-key 
--endpoint 
putF 


./splunk cmd splunkd rfs -- --access-key --secret-key 
--endpoint 


Ils --starts-with 
size,name 
are) e COREE 
Poey 





NOTE: You will see files other students have uploaded in the <s3_bucket_ns>/al1l1 directory. 


Look for your own test file. If you don't see your file, check splunkd-utility.log in 
sh5/var/log/splunk for any errors. Check your CLI attribute values carefully and try again. 
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Task 2: Test the SmartStore configuration on a test instance. 


4. To specify the SmartStore settings, edit ~/sh5/etc/system/local/indexes.conf and restart. 


NOTE: In this lab environment, you will uniquely identify your remote storage namespace using your 
student ID. Re-compose your path by replacing all with test-<x> for the indexes. conf 
settings. For example: path = s3://1sd1234/test-99 


vi ~/sh5/etc/system/local/indexes.conf 
[default] 

eSMOrekench = voline: sovol Lae nemne 
maxGlobalDataSizeMB = 500000 
maxDataSize = auto 


[volume:s3vol1 |] 
SEOMace lk pe = oee 
path = 
SIMONE 43S CCOO Ike 
re NOCA oSSCeSe a 
remote.s3.endpoint = 


./splunk restart 





NOTE: This configuration sets all indexes to use the SmartStore volume s3vol. It limits the size of 
each index shared across all peers to 500 GB (maxGlobalDataSizeMB). Upon restart, the 
cache manager uploads qualifying buckets and their metadata to the remote storage. 
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5. To confirm SmartStore is working, run the rfs command to list the buckets from your test instance. 


Compare the GUID of the instance against the remote bucket namespace. 


cat ~/sh5/etc/instance.cfg 
[general ] 
guid = BF18D6D8-3193-49C9-82C0-E72B56A87548 


./splunk cmd splunkd rfs -- ls --starts-with volume:s3vol 


size, name 

a a e 0 SS De Dios Oa CIs OE E a a a a 
BF18D6D8-3193-49C9-82C0-E72B56A87548/.rawSize 

sists. // aubyslaite, cls) Sel 16O,/ OIE Ie Deis = 3 LY sian COS 2008 | 225 aA | Sais / eibaelsye lpia 


BELS DOD = 31 S54 81 = 3 2CO=8, 7) ZB SGiNs 1543/ LaSo20S6e4 C= LS a 919262 0— 

LASS SI SOU SS SSS ZINSZ ss) 5). Sale 

Sor eincabie/ clo Sel ice UIE LoDo =m LSA CURB 200 8 R ee 7 Sais) emacs lla 
BF18D6D8-3193-49C9-82C0-E7/2B56A87548/Hosts.data 

Lily eiicliie/ clo Sel sce OILS DDOS Leis“ See 3 ACO 12 3S 6iNs 7 Seis a Ibi — 
BF18D6D8-3193-49C9-82C0-E72B56A87548/SourceTypes.data 

LOU; euch clo, Sel) iG) OSes DED oS Ls a ICC 6 COSI IZ. 3S GINs 7] Sis / a a a 
lol SDS DS T | GS=ACC Is ACUSIh | Za Gives (Ss / SouieGSes Celtel 

Zora enolic / clo Sc 6G) OIE IMDS Dis = BLS Sioa ICI 8) 260 | 28 ales o e oa usa 
BF18D6D8-3193-49C9-82C0-E72B56A87548/Strings.data 


NOTE: The rfs command is now using the values provided in the indexes. conf and recursively 
lists all Splunk buckets present in the remote storage. If you see the list of index files, your 
SmartStore configuration is set correctly. 


6. Log into https://{Public_DNS}/sh5. 


7. Runa search to confirm the CacheManager activities (Last 15 minutes): 


index=_internal sourcetype=splunkd component IN(CacheManager*) | table _time, 
action, status, cache_id 


To verify that the cache manager is functioning properly, select Job > Inspect Job. 


NOTE: Scroll down the Execution costs list and locate: 


command.search. index. bucketcache.error 
command. search. index. bucketcache.hit 
command. search. index. bucketcache.miss 


The listing of these components and their invocation counts indicate functioning SmartStore 
configuration. 
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Task 3: Run the migration on the indexer cluster. 


8. Access the Splunk Web interface for cmanager: https://{Public_DNS}/cmanager 


Confirm that the Health Status indicator is green. 


9. To convert any preexisting, single-site buckets to follow the multisite replication and search policies, 
edit ~/cmanager/etc/system/local/server.conf. 


[os _user@ip-10-0-x-2 bin]$ 


exit 


[os_user@ip-10-0-x-3 ~]$ 
vi ~/cmanager/etc/system/local/server.conf 


els EC rA] 


mode = master 

pass4SymmKey = $7SSbhODPOG£ZOWHE3/2KTBWNbiIVOEACQOhIVILPnBcP8F3TykW3d8znQ5pb 
multisite = true 

Wj WVES EMCI RelGeowe = IL 

SEeicCl eco = Il 

avellelole SleSss = Sire), Sire 

SUIS TS ae ea maGrem = @reuC ais Lp eOceul < 2 
SIIES SSE iG, meow O T ly ices 2 
MAMIGCSMSWSS MNCs = irelSe 

Giltsiccie mabe =] mexe os) Usioic 

ik Ouetwetecleie SIS ewe wee = Sa cS ls smiceZ 
constrain singlesite buckets = false 


~/cmanager/bin/splunk restart 





10. Log back into cmanager and verify again the status of the cluster. 


The manager node Health Status indicator may be in red initially. Be patient. Wait for it to turn green. 
11. Access the Indexer Clustering page and confirm that the cluster is in the complete state. 
12. Click the Indexes tab > the Bucket Status button, and confirm no buckets are stuck in fixup tasks. 


NOTE: Buckets can be stuck in fixups for various reasons. If you have buckets that are stuck in this 
exercise, click the Action button and delete them. 


13. Edit the ~/cmanager/etc/master-apps/_cluster/local/indexes.conf file with the SmartStore 
settings provided. 


Once again, you will uniquely identify your remote storage namespace. Re-compose your path by 
replacing test-<x> with idxc-<x> in indexes.conf. 
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vi ~/cmanager/etc/master-apps/ cluster/local/indexes.conf 
[default] 

TenotePatda = wolluimesSswol/e iicles< nenie 
maxGlobalDataSizeMB = 500000 

maxDataSize = auto 


[volume:s3vol1 |] 

SCO e elte = ceiloce 

eena == So) a a 

PONOC 5 SS Ce e e E Dima Gia Iain (OCIS Sak 
POMOCA oR S Cere Ie = sles Mela milo E TTM Keroro neve REUN 
remote.s3.endpoint = https://s3.ap-northeast-l.amazonaws.com 





NOTE: In production, the migration process will take a while to complete. If you have a large 
amount of data, expect some degradation of indexing and search performance during the 
migration. Schedule the migration for a time when your cluster activities will be idle. 


WARNING _ Don't forget to adjust the maxGlobalDataSizeMB and frozenTimePeriodInSecs settings 
to avoid unwanted bucket freezing and possible data loss. 


Remember SmartStore bucket-freezing behavior is different from the non-SmartStore 
behavior. 


14. From the manager node, run splunk validate cluster-bundle to check for any errors. 
15. If no errors are reported, deploy the bundle to the peer nodes: splunk apply cluster-bundle 


~/cmanager/bin/splunk validate cluster-bundle 
Created new bundle with checksum=C2 6EAFY9A245D6E8C649316082D0896D9 


~/cmanager/bin/splunk show cluster-bundle-status 
~/cmanager/bin/splunk apply cluster-bundle 


~/cmanager/bin/splunk show cluster-bundle-status 
el SCS Te 


eilus ter ete e e= e Mesceimc Cir chiS Bees aS) id) Pe OC mSss : 





Wait for the rolling restart to complete. 


Check Your Work 


Task 4: Confirm remote storage access across the indexer cluster. 


16. Verify the status of the cluster from cmanager's Splunk Web. 


© 2021 Splunk Inc. All rights reserved. Splunk Cluster Administration - Lab Exercises February 25, 2022 62 


splunk > 


The Health Status indicator is still green, and the cluster is still complete. This may take some time. 
17. Run a search to monitor the migration process: 


| rest /services/admin/cacheman/_ metrics splunk_server=idx* | fields 
splunk_server migration. * 


18. Run a search to check if the migration was successful: 


| rest /services/admin/cacheman splunk_server=idx* | search cm:bucket.stable=0 | 
stats count by splunk_server 


NOTE: When migration.status in search from #17 returns finished on all peers and the result of 
search #18 is no results found, the migration is complete. 


At this point, you should be able to run normal searches without much delay because the 
data is already in the local cache. 


19. In dserver's Monitoring Console, navigate to Indexing > SmartStore > SmartStore Activity: 
Deployment. 


20. Select the Enable check box under Show Migration Progress and verify that the remote storage is 
ONLINE and the migration progress is at 100. 
21. Run a search to confirm the CacheManager activities (Last 15 minutes): 


index=_internal host=idx* sourcetype=splunkd component IN(CacheManager*) | table 
_time, action, status, cache_id, splunk_server 


To verify that the cache manager is functioning properly, select Job > Inspect Job. 


NOTE: Scroll down the Execution costs list and locate: 


command.search.index.bucketcache.error” - - - 
command.search. index. bucketcache.hit 12 = - 
command.search.index.bucketcache.miss - - - 


The listing of these components and their invocation counts indicate a functioning 
SmartStore configuration. Ideally, you want the results from only the 

command .search.index.bucketcache.hit invocation. 

If not, wait a little bit and re-run the search. 
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Troubleshooting Suggestions 

If your configuration is not returning the expected results, troubleshoot by isolating the issue. 

1. Verify the command syntax and spelling on each instance with: splunk btool check --debug 
2. Carefully review the values specified in ~/cmanager/etc/system/local/server.conf. 


3. Change the log level for S3Client and StorageInterface to DEBUG and check splunkd.log and 
splunkd-utility.log for any error details. 
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